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ABSTRACT 



Method and apparatus for evaluating the quality of a packet- 
switched voice connection. The apparatus includes measure- 
ment circuitry coupled to receive a voice signal. The mea- 
surement circuitry measures rate of packet loss and at least 
one other objective characteristic of the voice signal. The 
rate of packet loss and the at least one other objective 
characteristic are related to a plurality of quality character- 
istics affecting the quality of the voice signal as perceived by 
users, such that measurements of the rate of packet loss and 
the at least one other objective characteristic can be trans- 
lated into subjective quantifications of each of the quality 
characteristics. A database stores an effects matrix. The 
effects matrix provides quality information for voice signals 
that include different combinations of subjective quantifica- 
tions for each quality characteristic. Software operating on 
the apparatus utilizes measurements of the rate of packet 
loss and the at least one other objective characteristic and the 
effects matrix to generate quality information for the voice 
signal. 

28 Claims, 5 Drawing Sheets 
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METHOD AND SYSTEM FOR EVALUATING 
THE QUALITY OF PACKET-SWITCHED 
VOICE SIGNALS 

CROSS-REFERENCE TO RELATED 
APPLICATIONS 

The following copending commonly assigned U.S. patent 
applications are hereby incorporated herein by reference: 
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Attorney 


Ser. No. 


Filing Date 


Docket No. 


08/751,023 


November 18, 1996 


VON-96-001 


08/798,350 


February 10, 1997 


VON-97-004 



FIELD OF THE INVENTION 

This invention relates generally to telecommunications 
and specifically to a method and system for evaluating the 
quality of packet-switched voice signals. 

BACKGROUND OF THE INVENTION 

Telephone connections have always been subject to 
impairments in the form of noise, attenuation, distortion, 
crosstalk, and echo. Such impairments are particularly com- 
mon to analog portions of the connection, such as along 
subscriber loops and within frequency domain multiplexing 
equipment. Digital transmission alleviates many of these 
problems but introduces quantization noise or glitches. Even 
with perfect digital transmission applied for long-haul 
connections, a typical telephone connection includes many 
analog components, such as hybrids, where impairments can 
occur. 

A poor connection or a malfunctioning piece of equip- 
ment can produce conditions that a telephone customer will 
find objectionable or intolerable, so that the connection is 
deemed to be of very poor quality. When there is a high 
incidence of such poor connections, customers may com- 
plain to the service provider or regulatory authorities, or 
simply change long distance carriers. Perceived quality of 
telephone connections is therefore a major factor affecting 
the reputation and marketability of long distance telephone 
services. 

To guard against poor quality, telephone service providers 
have developed methods to lake objective quality measure- 
ments upon a line, a piece of equipment, or even an 
end-to-end telephone connection. These measurements can 
help the service provider detect and gauge impairments, 
pinpoint weak elements and correct deficiencies that degrade 
user perception of quality. 

Many such objective measurements are well known and 
standardized in the art. Empirically-derived thresholds 
enable analysts to infer the existence and severity of quality 
problems by comparing measurements to tables of accept- 
able value. For example, power levels of test signals and 
quiet channel noise can be measured electronically. Since it 
is well known that a certain range of signal levels must reach 
a telephone receiver to produce acceptable volume at the 
earpiece, and C-weighted noise must be kept at a level 
relative to the signal level to keep users from experiencing 
unacceptable noise at the earpiece, the combination of 
measures for a particular connection supports accurate 
assessment of the likelihood that a user would find that 
connection to be of poor quality with respect to ability to 
hear the distant talker. 
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For such objective measurements, the effect of extreme 
values on user perception of quality is clear, and there are 
easily discerned thresholds for "no effect" and "substantial 
degradation" conditions. However, for immediate values, 

5 there is generally no clear division between values repre- 
senting acceptable and unacceptable connection quality. 
Speech clarity and perceived connection quality depend on 
many variables, including, for example, speech content, 
talker rhythms, subjective perception of the listener, and 

3Q users' acclimation to their telephone service. As a result, the 
correlations between values of objective measures and user 
perception of connection quality are statistical, representing 
the combined effects of many different kinds of impairments 
and variations of sensitivity to them among the population. 

15 Earlier work by the inventor and others in this area have 
created mappings between objective measurements and per- 
ceived quality, so that, for example, when a given circuit was 
measured in terms of signal level, noise, distortion, 
crosstalk, and echo, the mapping predicts the percentage of 

2Q conversations that would be reported as being significantly 
impaired or of poor quality as perceived by an average user 
population. Such mappings have proven to be a powerful 
tool for analyzing reported impairments and for gauging 
acceptable performance of a new line or piece of equipment 

25 before deployment. 

The mapping was produced by creating or finding various 
combinations of measurable characteristics along telephone 
circuits and then having a population of callers conduct test 
calls to subjectively gauge the quality of each call. For each 

30 test call, the circuit under analysis was rated on a scale of 
None-Some-Much for each of the impairments manifested 
to users of the connection related to the selected objectively 
measurable characteristics. These impairments include 
noise, volume, distortion, and echo. Each caller also pro- 

35 vided an opinion score, which is an overall rating of the 
circuit quality on a numerical scale. 

Each caller also determined whether the overall effect of 
the impairments was to render the connection: unusable (U; 
rendering the channel entirely unusable), difficult (D; caus- 

40 ing enough difficulty to require adaptation by the speaker 
and listener), irritating (I; disturbing but not requiring adap- 
tation by the speaker and listener), noticeable (N; minor 
enough to be ignored), or unnoticeable (O; no effect on 
quality). The percentage of calls or connections that elicit 

45 any one of the first three responses (unusable, difficult, or 
irritating) is called the P(UDI). The P(UDI) is of particular 
interest to service providers as a meter of customer satis- 
faction because it has been shown that overall satisfaction 
decreases as P(UDI) increases, regardless of average opin- 

50 ion score. 

Analysis of empirical data including user reports of 
impairments and perception quality, together with user 
reports of impairments obtained in conjunction with objec- 
tive measurements of connection characteristics then sup- 

55 ported a two step development of a means for predicting user 
perception of quality from objective measurements. First, a 
model supporting prediction of P(UDI) and average opinion 
score as a function of percentages of calls with each of the 
possible combinations of "none," "some," and "much" (N, 

60 S, M) conditions reported for each of the impairments 
considered was produced. Then, objective measurements 
were correlated with user reports of impairments to predict 
the proportion of N, S, M reports from users that will 
eventuate as a function of objective measurements. From 

65 these two elements, it was then possible to take measure- 
ments of the objective characteristics for connections and 
translate the set of measures obtained into estimates of likely 
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user perception of quality as revealed by P(UDI) and the (e.g., a line or a piece of equipment) can be tested before 
average opinion score. being used commercially. In another application, a worksta- 
tion stores and analyzes objective measurements. This work- 
SUMMARY station can be used by an engineer troubleshooting a net- 
In one aspect, the present invention provides a technique 5 work or a sales engineer comparing qualities of various 
for assessing the quality of a packet-switched communica- options. The present invention could also be used in an 
tions channel. For instance, the technique provides a means automatic, real-time control system for a communications 
by which a set of objective measurements for a packet- network. 

switched telephony connection can be processed to derive a DDICC nccpDiDTrnM m? ttjc rn> awimpo 

. , , / , , i r *u .* i* 10 BRIEF DESCRIPTION OF THE DRAWINGS 

projected perceptual quality level for the connection. It IU 

extends upon the techniques of the prior art to adapt for the FIGS, la and lb are examples of two communications 

effects of important new technologies. networks that can utilize the present invention; 

Specifically, the present invention addresses packet- FIG. 2 is a diagram of a process that converts objective 

switched (e.g., Internet Protocol based) telephony, which is measurements into estimates of user perception of quality 

subject to a different set of impairments from ordinary 3 for packet-switched telephony; 

link-switched telephony. The packet-switched environment F]GS 3a _ 3c m waV eforms illustrating the effect of 

can produce momentary interruptions or latencies in a packet loss on a pac k e tized voice signal; 

streaming signal. In addition some speech compression and nQ 4 fa a ^ d of communication network lhat 

coding schemes can suffer dropouts or garbled portions ^ invent 

depending upon the loss of certain packets. Such impair- anc j 

ments represent new phenomena that must be incorporated 

into the measures-to-impairments and impairments-to- FIG * 5 » a block dia S ram of aD apparatus that can be used 

quality transforms used to predict user perception of quality. 10 evaluate a communication channel. 

In one embodiment, the present invention provides a 25 DETAILED DESCRIPTION 
method of evaluating the quality of a packet-switched voice 

signal. To begin, a plurality of objective characteristics for The making and use of the various embodiments are 

a voice signal being transmitted across a packet-switched discussed below in detail. However, it should be appreciated 

network are selected. At least one of these objective char- that the present invention provides many applicable inven- 

acteristics is packet loss. For each of the objective tive concepts that can be embodied in a wide variety of 

characteristics, correlations are determined in order to cat- specific contexts. The specific embodiments discussed are 

egorize quantitative measurements of the objective charac- merely illustrative of specific ways to make and use the 

teristic into a number of categories, representing a subjective invention and do not limit the scope of the invention, 

assessment of the presence and severity of impairments as In a first aspect, the present invention allows for a 

perceived and described by users. To do so, a plurality of 35 correlation between objective measurements and perceived 

evaluation voice messages is generated by varying selected quality in the context of a packet-switched communications 

ones of the plurality of objective characteristics for each of channel. 

the evaluation voice messages. The quality of each of the FIGS, la and lb illustrate simplified block diagrams of 

evaluation voice messages is then empirically measured by two examples of networks that can utilize the present 

having a test subject (or subjects) listen to the evaluation 4Q invention. Referring first to FIG. la, a hybrid telephone 

voice messages and grade each voice message with respect network 10 includes both circuit switched network portions 

to the selected impairments. The probability that a packet- 14 and 22 and a packet-switched network portion 18. To 

switched voice signal will achieve a particular rating is then utilize the hybrid telephony network 10, an initiating tele- 

determined, by suitably amending extant effects models, phone 12 initiates a call with the circuit switched network 

which correlate subjective assessments of presence and 45 14. i n the preferred embodiment, the initiating telephone 12 

severity of impairments with subjective measures of overall is an ordinary telephone and the circuit switched network 14 

connection quality. The determination thus utilizes a mea- is the public switched telephone network (PSTN). The 

surcment of objectively quantifiable characteristics of the present invention, however, is equally applicable to use with 

packet -switched voice signal to predict the subjective user proprietary equipment and private networks, 

quality rating of the packet-switched voice signal. 5Q Circuil sw i tc hed network 14 routes the call to a gateway 

In one embodiment, the objective characteristics are cor- 16, sometimes referred to as an internet telephony gateway 
related with corresponding user perception of the severity of (ITG). Gateway 16 converts the circuit switched telephone 
the manifestation of that characteristic as an impairment to signals into packets that are compatible with the packet- 
conversation. For example, if a packetized voice signal is switched network 18. While many protocols can be utilized, 
missing some packets, a listener may perceive speech dis- 55 m0 st current networks carrying packet-switched voice use 
tortion. In this case, the objective characteristic is packet Internet Protocol (IP). Gateway 16 will also have access to 
loss and the manifestation of that characteristic to the user is a directory (not shown) of addresses (e.g. IP addresses for an 
speech distortion. IP-based network). Utilizing this directory, gateway 16 

The techniques of the present invention allow objective addresses the packets to an appropriate exit gateway 20. 
measurements to replace subjective measurements upon a $0 The voice signals are then transported through packet- 
channel. This yields an improvement, for example, in gaug- switched network 18 from gateway 16 to gateway 20. 
ing performance of a channel or a communications element Packet-switched network 18, sometimes referred to as a 
before il is put into service. The objective assessment of an converged network, combines various types of media, such 
Internet telephony channel is expressed in a way that is as voice calls, data, and streams of video, onto a single line, 
relevant to perceived quality. 65 All these different media are chopped into discrete blocks of 

The present invention can be used in a variety of contexts. data or packets. As mentioned above, packet-switched net- 

For example, a phone system or a portion of a phone system work 18 can be an IP-based network. 
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One example of an IP-based network is the public Inter- 
net. In this context, the "Internet" (uppercase "I") or public 
Internet is used to connote the worldwide collection of 
interconnected networks that uses Internet Protocol (IP) to 
link a large number of physical networks into a single logical 
network. Physically, the Internet is a huge, global network 
spanning nearly 100 countries and comprising a great num- 
ber of academic, commercial, government, and military 
networks. 

Packet-switched network 18 could also comprise other 
IP-based networks as well as other communication net- 
works. For example, packet-switched network 18 could 
comprise an internet which is not connected to the public 
Internet. In this context, an "internet" (lowercase "i") is any 
collection of separate physical networks, interconnected by 
a common protocol, to form a single logical network. An 
internet would preferably, but not necessarily use Internet 
Protocol. An internet which is owned by a single entity is 
sometimes referred to as an intranet or virtual private 
network. Network 18 can comprise an intranet, which is or 
is not connected to the Internet. 

For final delivery, gateway 20 converts the packets back 
into a format that is compatible with circuit switched net- 
work 22. As before, in the preferred embodiment circuit 
switched network 22 is the PSTN. In other words, circuit 
switched network 14 and circuit switched network 22 may 
be two portions of the same network, for example in two 
different parts of the country or the world. The telephone call 
can then be routed to receiving telephone 24. 

The telecommunications network of FIG. lb is similar to 
that of FIG. la except that it is an end-to-end packet- 
switched network. In this case, the initiating telephone 12' 
can generate packet-switched signals and provide these 
directly to packet-switched network 18'. In this embodiment, 
the packets can be generated either by telephone 'or by 
equipment (not shown) that is relatively close to telephone 
12* (for example, in the same building). The call Ls routed 
across packet-switched network 18' to receiving phone 24'. 

Further details on hybrid and end-to-end packet -switched 
networks are provided in commonly assigned patent appli- 
cations Ser. No. 08/751,023 filed Nov. 18, 1996 (Attorney 
Docket No. VON-96-001) and Ser. No. 08/798,350 filed 
Feb. 10, 1997 (Assignee Docket No. VON-97-004), each of 
which is incorporated herein by reference. 

In one aspect, the present invention relates to a method for 
evaluating the quality of a packet-switched voice signal (or 
connection). A preferred embodiment flow diagram of one 
such method is illustrated in FIG. 2. The voice signal can be 
transmitted along the network 10 illustrated in FIGS, la or 
lb or along portions thereof. For example, a telephony 
service provider may wish to test a new piece of equipment 
or a new trunk prior to placing it on the network. 

To begin, objective characteristics of the packet-switched 
voice signal under test are selected. These objective char- 
acteristics are measurable and can be used to characterize the 
signal. Objective characteristics include signal loss, noise, 
waveform distortion, packet loss, echo path delay, and echo 
path loss as illustrated by test matrix 210. Some of the 
attributes that are measured may be common with those 
measured for satellite or terrestrial networks. If this is the 
case, data derived from measurements and studies of these 
networks can be used along with the measurements and 
studies performed on a packet-switched network. 

Some of the objective characteristics are unique to packet- 
switched systems. For example, packet loss is a measurable 
attribute of a packet-switched voice signal. As it turns out, 
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in performing experiments in work leading to the present 
invention, the inventor has discovered that packet loss tends 
to be an important objective characteristic for many packet- 
switched telephony systems. 
5 Each of the objective characteristics will lead to a mani- 
festation in the perceived quality of the voice signal. Table 
1 provides a listing of the manifestations corresponding to 
selected measurable characteristics. Other characteristics 
could also or alternatively be utilized. 
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TABLE 1 



15 



Objective Characteristic 


Manifestation 


Loss (or level) 


Low volume 


Noise 


Noisiness 


Waveform distortion 


Speech distortion 


Packet loss 


Speech distortion 


Echo path delay 


Echo 


Echo path loss 


Echo 



20 The level for each objective characteristic is transformed 
into an expected percentage of the population that would 
categorize the manifestation into one of a number of sub- 
jective quantifications. Different manifestations (or quality 
characteristics) can have different numbers of subjective 

25 quantifications. For example, in the preferred embodiment 
three subjective quantification categories — none, some and 
much — are defined for each quality characteristic. Test sub- 
jects are then instructed to rate the presence of an impair- 
ment as "none" if it is absent, "much" if it is present and very 

30 noticeable, and "some" otherwise. 

Subjective evaluators are used to determine the correla- 
tion between objective measurements and none-some-much 
ratings. Typically a large number of subjective evaluators 
(e.g., 44 people or more) are used to evaluate the system. 

35 These subjective evaluators listen to test messages and rate 
the effect of the objective characteristic on the manifestation. 
For example, the evaluator will listen to a test signal or series 
of test signals with varying signal noise (an objective 
characteristic) and then describe the signal as having "none", 
"some" or "much" noisiness (a quality characteristic). The 

40 same tests are performed for the other objective character- 
istics to determine the perceived effect on the voice signal. 
These tests can be performed by varying one characteristic 
at a time or by varying more than one objective character- 
istic. The evaluator may (or may not) also be asked to 

45 evaluate the effect on the voice signal and the overall quality 
of the message. 

Evaluation voice messages are generated by varying 
selected ones of the objective characteristics. For example, 
a first evaluation message may have 35 dBrnc noise, -25 dB 

50 loss, packet loss at a rate of one packet per minute, echo path 
loss of -25 dB, and an echo path delay of 20 msec. Other 
evaluation messages would have a different variation of 
these characteristics. Messages within the same classes can 
be duplicated with different voice content. In this manner a 

55 set of evaluation messages is generated. 

These other evaluation messages are designed to represent 
variations with respect to each objective characteristic over 
the range between values which represent levels of which 
nearly one hundred percent in a population would report the 

60 associated impairments as "none" and "much". 

The evaluation voice messages can be generated prior to 
empirical testing or can be real-time telephone conversa- 
tions. Prior generated messages are useful because param- 
eters can be precisely varied in the laboratory before hand 

65 and the content can be held constant. Real-lime conversa- 
tions are useful because parties on either end can evaluate 
the call. 
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The data collected from the evaluation messages then the presence and severity of selected impairments and 

provide a basis for estimating the percentage of "none", effects and/or quality measures for each call as described 

"some", and "much" responses as a function of the values of above. 

the objective measurements, thereby supporting develop- As can be seen, this aspect of the present invention 

ment of ihcx^ transform in FIG. 2. The x 2 transform is used 5 provides a process that correlates quantifiable parameters 

to translate measurement of each of the objective charac- with quality characteristics (i.e., impairments) and correlates 

teristics into subjective quantifications of each of the quality these quality characteristics with user perception. In this 

characteristics. For example, FIG. 2 illustrates how the two-step calculation process, objective measurements can be 

measurements of objective characteristics such as loss translated into estimates of the P(UDI) and expected opin- 

(PWR), noise (NSE), echo path delay (EPD), echo path loss 10 10Q ^ scorc for a Pf 1 ™^ f™*-. 

(EPL), and packet loss rate (PLR) can be used to determine Continuing with FIG. 2, the input data are a set of 
subjective quantifications (e.g., none, some, much) of qual- measurements acquired via repetihous testing of 
ity characteristics such as low volume, noise, echo and connections to produce a set of measurements that fill in the 
J . . objective measurements matrix 210. These data are then 
speec is or ion. translated via a first transformation ^ into an impairments 
The overall quality of the evaluation messages can be " matfix 22Q ag discussed above nc entries of matrix 2 20 
empirically measured. In the preferred embodiment, a large constitute the proportion of calls for which the average user 
number of evaluation messages are played to a number of population would describe the presence of each of a par- 
test subjects who subjectively evaluate and rate the quality ticular set of impairments identified for those users as being 
of each. For example, the test subjects use a scale of one to 'none' 'some*, or 'much'. 

five to rate each message. The ratings ;could be unusable(U; 20 ^ impairments matrix 220 is then translated via a 

rendering the channel enttrely unusable), difficult (D;caus- smad transform ^ by which an effects model 2 20 is 

ing enough difficulty to require adaptation by the speaker applied lQ the matrix t0 produce estimates of 

and listener), irritating (I; disturbing but not requiring adap- the avefage p(UDI) and meaQ Qp{nioQ ^ The effects 

tation by the speaker and listener), noticeable (N; minor model provides an estimate for eacfa of the possible none/ 

enough to be ignored), or unnoticeable (O; no effect on 25 combinalions of the impairments in the impair- 

quality). Alternatively, voice quahty can be graded on a scale mems matrix of the associated values of P(UDr) and mean 

of bad (B), poor (P), fair (F), good (G), and excellent (E). opmioQ mn (MQS) 

As further objective measurements upon the packet- j t ^ contemplated that some savings may be achieved by 

switched channel, the frequency and duration of dropouts applying empirical knowledge from satellite or terrestrial 

due to lost or severely delayed packets are measured and communications. For example, the inventor has performed 

summarized statistically. Averaged dropout occurrence is eaflier experiments that have developed data that can be 

then used to calculate the probability that speech will be used in developing the transforms t, and and effects 

perceived as being distorted by sudden changes in speech matrix 2 30. Even through this data was not developed using 

waveforms or occurrence of incomplete words, when signal a packet-switched network, it can still be applied to a 

loss spans the duration of one syllable or more. packet-switched network. 

The empirical data is captured and summarized to map the Th e inventor has also learned voice signals transmitted 

combined effects of objective characteristics such as noise, v j a a packet-switched networks can suffer from speech 

signal level, distortion, and echo into a perceived quality distortion due to packet loss. Earlier work on voice over 

level. Because this mapping is independent of the underiy- ^ satellite introduced a need to characterize momentary 

ing transport, figures taken from, for example, common dropouts, along with noise and crosstalk. In the present 

analog telephony may be applied here as well. However, if invention, this mapping of a None-Some-Much to a P(UDI) 

the circuit under test differs substantially from the circuit ^ applied to the more current problem. The occurrence of 

used to generate the data, then a new characterization is incomplete words projected for a packet-switched call is 

required. 45 treated as if it were an objective measurement provided to 

The test data can also be used to develop a mean opinion the model used for voice over satellite. This yields a P(UDI) 

score (MOS) for each of the combinations of subjective contributed by the long duration dropouts, 

quantifications. A mean opinion score provides the average Dropout statistics are also used to calculate a probability 

score for all (or most) subjective evaluators. For example, a 0 f s h 0 rt dropouts that cause distorted waveforms, or a 

system can be used where the B (bad), P (poor), F (fair), G 50 so-called "click/warble" effect. This new form of distortion 

(good), and E (excellent) are rated at 0, 1, 2, 3, and 4, can be correlated with an objective measurement of the rate 

respectively. Accordingly, a perfect score would be 4.0 (e.g., 0 f loss of packets. FIGS. 3a-3c illustrate an example of the 

all evaluators provide a rating of "E" or four points). e ff ec t of packet loss on a voice signal. FIG. 3a illustrates a 

In the preferred embodiment, the BPFGE MOS is used voice signal with no appreciable packet loss ("none"). FIG. 

along with the probability that the line is acceptable P(UDI). 55 3b illustrates the same voice signal with packet loss at the 

The combination provides valuable information because, for rate of 5 packets/minute while FIG. 3c illustrates the voice 

example, where an MOS of 3.6 is very good and might signal with packet loss at the rate of 15 packets/minute. The 

normally be considered to guarantee user satisfaction, stud- effect of these different levels can be correlated with the 

ies have shown that users will nonetheless complain of poor subjective perception of speech distortion in a voice signal, 

quality when the P(UDI) is greater than six percent regard- $o to surmise that, absent other gauge of speech distortion, 

less of the MOS. Therefore, the overall service predictions nearly all users will report speech distortion as "none" for 

are best made when using both MOS and P(UDI). the case of FIG. 3b and "much" for the case of FIG. 3c. 

The probability that a packet-switched voice signal is Further empirical testing will reveal the percentage of 

acceptable is determined by application of a second trans- "none," "some," and "much" responses for the case of FIG. 

form and use of an effects matrix 230, as illustrated in 65 $b. 

FIG. 2. Effects matrix 230 can be produced from an appro- The present invention can be utilized in a variety of 

priate data in which users have reported for any calls both contexts. For example, in one embodiment, it is contem- 
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plated to repeat the empirical experiments to account for Computer apparatus 540 runs software that calculates the 

new effects in the packet-switched realm, new spoken lan- quality information for a given input voice signal. This 

guages or coding schemes wherein some impairments may software utilizes measurements of the objective 

have a greater or lesser effect, or new channel bandwidths. characteristics, including packet loss and the effects matrix 

For example, empirical test results may be different for 5 to generate quality information for the voice signal. For 

voice signals spoken in English as opposed to Chinese. As example, the software would preferably translate the objec- 

a result, a network in the United States may be optimized tive characteristics into a specific combination of subjective 

differently than one in China. When setting up an interna- quantifications. It would then translate the specific combi- 

tional calling plan, the user could specify a language and nation via the effects matrix. 

calls could be routed to optimize the quality for this lan- 1° Once the quality information is calculated, it will be 

guage. This specification can be for all calls or include utilized by output circuitry 550. The output circuitry 550 

different optimizations depending upon the telephone num- will depend upon the specific application. In an engineering 

ber on the other end. test set, for example, output circuitry 550 may comprise a 

In one aspect, the present invention solves the problem of visual display or a file (e.g., in system memory or on a disk), 

how to measure and express the quality of a packet-switched 15 Output circuitry 550 could also comprise a port where the 

connection and especially, how to predict the callers' data is transported from computer 540. 

responses to a prospective type of connection or component In commercial network evaluation applications, as 

before it is actually released into use. This technique can be another example, output circuitry 550 could comprise a 

used in a variety of applications. connection to real-lime control circuitry (e.g., as could be 

One embodiment is envisioned as a workstation, perhaps 20 used in the example of FIG. 4). Alternatively, or in addition, 

even coupled to a network of interest, wherein objective output circuitry could provide a service message to a 

measurements are stored and analyzed according to the technician, for example, by pager, e-mail, audio display 

process of the present invention. An engineer looking for and/or visual display. Other examples exist. Certainly any of 

problems areas within the network, or a sales engineer the examples provided with the engineering test set example 

comparing qualities of various options, may access the 25 would also apply in commercial network evaluation 

workstation to perform projections applicable to some or all applications, and vice versa. 

of the network. Yet another application of the present invention is with the 

Another embodiment is envisioned as test equipment that use of a real-time control system for a telephone network, 

performs measurements or receives data pertaining to a 3Q Such a system is described with reference to FIG. 4, which 

channel under test and then applies the above process to illustrates a hybrid network similar to the one described with 

output a P(UDI) and MOS as a figure of merit for the respect to FIG. la. The control system would work equally 

channel under test. The test equipment might also perform a as we H with an end-to-end packet-switched telephony sys- 

sensitivity analysis to identify which of the objective mea- tem such as the one shown in FIG. 16. 

surements should be adjusted to best improve the perfor- 35 To begin, it is understood that the block diagram of FIG. 

mance of the channel. Without the present invention, an 4 (as well as FIGS, la and lb) is a gross over-simplification 

engineer might be inclined to make a decision based upon a of what a "real world" network would look like. Much 

noise parameter whereas the more significant a factor might equipment is not illustrated for the sake of simplicity. For 

be distortion due to dropouts. example, the routing within the packet-switched network 18 

As an example, FIG. 5 illustrates a system 500 for 40 is illustrated by three routers 30, 32 and 34. In any practical 

evaluating the quality of a packet-switched voice channel. A system, more routers as well as other components would be 

receiver 510 is coupled to the packet-switched voice channel used. 

(not shown). Receiver 510 can, for example, be coupled a As discussed above, a phone call is routed between 
router (see routers 30, 32, 34 in FIG. 4), a gateway (see gateway 16 and gateway 20 (either 20a or 2i)b) so that users 
gateways 16, 20 in FIG. la)> or another portion of the 45 at telephones 12 and 24 may carry on a conversation. In 
channel. Alternatively, the receiver can be coupled to a particular, gateway 16 is coupled to a router 30. Router 30 
specific piece of equipment which in not coupled to a reads the address on the packets and sends them to router 32 
network. In any event, the receiver 510 will receive a voice (via any of a number of pathways which may vary between 
signal from the packet-switched voice connection. packets). Router 32 is coupled to gateway 20b which pro- 
Measurement circuitry 520 is coupled to the receiver 510. 50 y ides the signal to the circuit switched telephone network 22 
The measurement circuitry 520 measures objective charac- and then to telephone 24. 

teristics of the voice signal. Preferably, one of these objec- Detection circuitry 36 is coupled to router 32 and is used 

tive characteristics is the rate of packet loss. As discussed to measure objective characteristics of the packet-switched 

above, each of the objective characteristics is related to a signal at router 32. For example, detection circuitry 36 may 

plurality of quality characteristics affecting the quality of the 55 be software running on a computer (e.g., the same computer 

voice signal as perceived by users. Using techniques which serves as router 32) to measure the number of packets 

described above, measurements of the objective character- lost at the router. Based on the packet loss, and possibly 

istics can be translated into subjective quantifications of other objective characteristics, the detection circuit calcu- 

each of the quality characteristics. lates the P(UDI) of the signal. If the P(UDI) goes above a 

The system 500 also includes a database 530 that stores an 60 certain threshold, the detection circuit will cause a message 

effects matrix (see element 230 of FIG. 2). As before, the to be sent to router 30 that will cause the signals to be 

effects matrix provides quality information for voice signals rerouted away from the troublesome portion of the network, 

that include different combinations of subjective quantifica- In the illustrated example, router 30 will re-route the voice 

tions for each quality characteristic. The quality information traffic to router 34, which is coupled to gateway 20a. 

preferably includes both mean opinion score (MOS) data 65 Detection circuitry 30 can be incorporated in many dif- 

and a probability that the voice signal is of an acceptable ferent portions of the network. In this manner, the source of 

quality level (P(UDI)). errors can be detected. With this information, calls can be 
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re-routed and service personnel can be notified. For 
example, an automatic message, e.g., visual indication, 
pager, e-mail, can be initiated when an error is found. Using 
a system, such as the one described here, would lower the 
number of dissatisfied customers by predicting and correct- 5 
ing low quality voice connections. 

As yet another example, the present invention can used to 
aid in the design of network components. Since the inven- 
tion provides a mathematical model, an engineer can predict 
what would happen if the characteristics of one of the 30 
components were to change. For example, an engineer may 
wish to lower the performance requirements of an echo 
canceler, to lower cost. Before building new equipment, the 
effect of such a redesign could be predicted and any desir- 
able design modifications made before finalizing the design, is 

While this invention has been described with reference to 
illustrative embodiments, this description is not intended to 
be construed in a limiting sense. Various modifications and 
combinations of the illustrative embodiments, as well as 
other embodiments of the invention, will be apparent to 20 
persons skilled in the art upon reference to the description. 
It is therefore intended that the appended claims encompass 
any such modifications or embodiments. 

What is claimed is: 

1. A method of evaluating the quality of a packet-switched 25 
voice signal, the method comprising: 

selecting a plurality of objective characteristics for a voice 
connection being transmitted across a packet-switched 
network, at least one of the objective characteristics 

30 

comprising packet loss; 

selecting a plurality of quality characteristics of a voice 
signal, each of the quality characteristics affecting the 
quality of the voice signal as perceived and described 
by users; 35 

generating a plurality of evaluation voice messages by 
varying selected ones of the plurality of objective 
characteristics for each of the evaluation voice mes- 
sages; 

developing a transform that can be used to translate 40 
measurements of each of the objective characteristics 
into subjective quantifications of each of the plurality 
of quality characteristics; 

empirically acquiring user perception data by having at 
least one test subject listen to the evaluation voice 45 
messages and rate the quality of each evaluation voice 
message as one of a plurality of subjective quality 
ratings and also having the at least one test subject 
provide an overall quality opinion score of each of the 
evaluation voice messages; 50 

developing an effects model based on the empirically 
acquired user perception data, the its effects model 
providing perceived quality as a function of the sub- 
jective quantifications of the quality characteristics; 
and 55 

determining a probability that the packet-switched voice 
signal will fall within a selected one of the subjective 
quality ratings by measuring each of the objective 
characteristics and then translating measurements of 
the objective characteristics into subjective quantifica- 60 
tions of each of the plurality of quality characteristics, 
the subjective quantifications being used with the 
effects mode! to determine the probability that the 
packet-switched voice signal will fall within a selected 
one of the subjective quality ratings. 65 

2. The method of claim 1 wherein in at least one of the 
objective characteristics is selected from the group consist- 



ing of signal loss, signal level, noise, echo path delay, and 
echo path loss. 

3. The method of claim 1 wherein developing a transform 
comprises developing a transform that can be used to 
translate measurements of each of the objective character- 
istics into one of three subjective quantifications of each of 
the plurality of quality characteristics. 

4. The method of claim 1 wherein the effects model 
includes data relating to the perceived quality of each of a 
plurality of voice signals, the voice signals having varying 
subjective quantifications of the each of the plurality of 
quality characteristics. 

5. The method of claim 1 wherein the method is used to 
evaluate a portion of a communications network. 

6. The method of claim 5 further comprising transmitting 
a voice signal across the portion of the communications 
network before the portion is placed into service. 

7. The method of claim 1 further comprising measuring 
objective characteristics of packet-switched voice signals 
being transmitted across a commercial network in real time. 

8. The method of claim 7 further comprising predicting 
the quality of the packet -switched voice signals being trans- 
mitted across the commercial network in real time. 

9. The method of claim 8 further comprising providing a 
quality control message if the quality predicted falls below 
a quality threshold level. 

10. The method of claim 9 further comprising rerouting 
the packet-switched voice signals in response to the quality 
control message. 

11. A method of controlling communications across a 
packet-switched network, the method comprising: 

receiving a voice message at a first portion of the network, 
the voice message being in packet form; 

routing the voice message from the first portion of the 
network to a second portion of the network; 

evaluating the voice message at the second portion of the 
network to determine packet loss and at least one other 
objective characteristic, the rate of packet loss and the 
at least one other objective characteristic being related 
to a plurality of quality characteristics affecting the 
quality of the voice signal as perceived by users such 
that measurements of the rate of packet loss and the at 
least one other objective characteristic can be translated 
into subjective quantifications of each of the quality 
characteristics; 

predicting a quality level of the voice message at the 
second portion of the network, the prediction being 
based, at least in part, on determined levels of packet 
loss and the at least one other objective characteristic, 
the prediction also being based on data from an effects 
matrix that provides quality information for voice sig- 
nals that include different combinations of subjective 
quantifications for each quality characteristic; and 

if the quality level falls below a threshold quality level, 
generating a message indicating a low quality voice 
message. 

12. The method of claim 11 further comprising causing 
the first portion of the network to re-route voice communi- 
cations in response to the message. 

13. The method of claim 11 wherein a service notice is 
generated in response to the message. 

14. The method of claim 11 wherein the quality level 
includes both a mean opinion score and a probability that the 
voice signal will be acceptable. 

15. The method of claim 11 wherein the communications 
network comprises a hybrid communications network that 
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includes a packet-switched network portion and a circuit 
switched network portion. 

16. An apparatus for evaluating the quality of a packet- 
switched voice connection, the apparatus comprising: 

a receiver coupled to the packet-switched voice connec- 5 
tion to receive at least one voice signal from the 
packet-switched voice connection; 

measurement circuitry coupled to the receiver, the mea- 
surement circuitry measuring rate of packet loss and at 
least one other objective characteristic of the voice 10 
signal, the rate of packet loss and the at least one other 
objective characteristic being related to a plurality of 
quality characteristics affecting the quality of the voice 
signal as perceived by users such that measurements of 
the rate of packet loss and the at least one other 15 
objective characteristic can be translated into subjec- 
tive quantifications of each of the quality characteris- 
tics; 

a database for storing an effects matrix, the effects matrix 2Q 
providing quality information for the voice signal that 
includes different combinations of subjective quantifi- 
cations for each quality characteristic; and 

software operating on the apparatus, the software includ- 
ing computer instructions utilizing measurements of 2 s 
the rate of packet loss and the at least one other 
objective characteristic and the effects matrix to gen- 
erate quality information for the voice signal. 

17. The apparatus of claim 16 wherein the software 
includes computer instructions for: 30 

translating the objective characteristics into a specific 

combination of subjective quantifications; and 
looking up the specific combination in the effects matrix. 

18. The apparatus of claim 16 wherein the quality infor- 
mation provided by the effects matrix includes both a mean 35 
opinion score and a probability that the call will be accept- 
able. 

19. The apparatus of claim 16 wherein in the at least one 
objective characteristics is selected from the group consist- 
ing of signal level, noise, echo path delay, and echo path 40 
loss. 

20. The apparatus of claim 16 wherein the apparatus is 
coupled to a commercial packet-switched voice network and 
receives voice signals being generated by a commercial 
subscriber in real time. 45 

21. A method of evaluating the quality of a packet- 
switched voice signal, the method comprising: 

selecting a plurality of objective characteristics for a voice 
signal being transmitted across a packet-switched 
network, at least one of the objective characteristics 50 
comprising packet loss; 
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generating a plurality of voice test signals, the test signals 
having varying quantities of packet loss; 

experimentally determining at least one threshold level to 
categorize quantitative measurements of packet loss 
into a plurality of categories, the categories being 
determined by having at least one evaluator rate the 
amount of speech distortion for each of the plurality of 
voice test signals; 

providing threshold levels for each of the other objective 
characteristics to categorize quantitative measurements 
of the respective objective characteristic into a plurality 
of categories; 

generating a plurality of evaluation voice messages by 
varying selected ones of the plurality of objective 
characteristics for each of the evaluation voice mes- 
sages; 

empirically measuring perceived quality of each of the 
evaluation voice messages by having at least one test 
subject listen to the evaluation voice messages and rate 
the quality of each voice message as one of a plurality 
of subjective quality ratings; and 

determining a probability that the packet-switched voice 
signal will fall within a selected one of the subjective 
quality ratings by correlating a quantitative measure- 
ment of the objective criterion of the packet-switched 
voice signal with said subjective quality rating to 
predict the subjective quality rating of the packet- 
switched voice signal. 

22. The method of claim 21 wherein in at least one of the 
objective characteristics is selected from the group consist- 
ing of signal level, signal loss, noise, echo path delay and 
echo path loss. 

23. The method of claim 21 wherein the method is used 
to evaluate a portion of a communications network. 

24. The method of claim 23 further comprising transmit- 
ting the voice signal across the portion of the communica- 
tions network before the portion is placed into service. 

25. The method of claim 21 further comprising measuring 
objective characteristics of the packet -switched voice sig- 
nals being transmitted across a commercial network in real 
time. 

26. The method of claim 25 further comprising predicting 
the quality of the packet-switched voice signals being trans- 
mitted across the commercial network in real time. 

27. The method of claim 25 further comprising providing 
a quality control message if the quality predicted falls below 
a quality threshold level. 

28. The method of claim 27 further comprising rerouting 
the packet-switched voice signals in response to the quality 
control message. 
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